[release-5.0] NO-ISSUE: Test and script fixes found during 5.0.0-rc.0 testing - #7327

The "Case Insensitive Log Levels" test fails on ARM when testing
TraceAll because klog verbosity 10 generates ~144K journal messages,
exceeding journald's default rate limit of 10K messages per 30 seconds.
The suppressed messages include the startup config dump line that the
test greps for, making the assertion impossible to satisfy.

Instead of globally disabling rate limiting in all test VMs, scope the
fix to the logging suite: disable rate limiting in suite setup and
re-enable it in teardown.

The drop-in write and journald restart are wrapped in `bash -c` so the
redirection and restart run as root under SSHLibrary's sudo=True (the
shell that expands `>` would otherwise run unprivileged), and `mkdir -p`
keeps the keyword self-contained rather than relying on kickstart
provisioning of /etc/systemd/journald.conf.d.

Co-Authored-By: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED

The Remove Namespace keyword ran `oc delete namespace` under Run With
Kubeconfig's 300s process timeout with allow_fail defaulting to False.
On ARM dual-stack, namespace garbage collection (pod teardown plus
finalizers, slowed by OVN reconciliation saturating the CPU) can exceed
300s, so the oc process was killed, returned non-zero, and cascaded into
a suite teardown failure that retroactively marked all passing tests as
failed.

Delete with allow_fail=True (matching the pattern already used across the
suites, e.g. standard1/kustomize.robot) so a slow or killed cleanup does
not fail the teardown, then poll until the namespace is actually gone. The
poll asserts a genuine NotFound rather than any non-zero exit code (which a
transient API error could also produce), so a subsequent test can reuse the
namespace name without a "being deleted" collision, bounded to 5m.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED

Release scenarios running upgrade paths with LVMS workloads followed by
full standard suites were hitting timeout limits under I/O contention
when many VMs boot and pull images in parallel. Set the greenboot
healthcheck timeout to 1200s (from 600s) and the robot framework timeout
to 60m centrally in ci_phase_boot_and_test.sh for all release scenarios,
and remove the now-redundant per-scenario overrides so the value lives in
a single place.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED

openshift#7298 reduced release-scenario VM disks from 30GB to 20GB. That is safe
for the lvms-standard/standard scenarios (they create a single 1Gi PVC),
but the ginkgo scenario runs the full storage spec suite which requests
several 1Gi PVCs concurrently. At 20GB the topolvm data VG only has
~420MiB free, so 7 storage specs fail with:

  ResourceExhausted ... no enough space left on VG:
  free=440401920, requested=1073741824  (arm-el10)

and on x86 el10 the same undersized VM shows etcd ReadIndex latency and
apiserver TLS-handshake flaps from I/O contention. Restore only this
scenario to --vm_disksize 30; the other nine 20GB scenarios stay as-is
since a single 1Gi PVC fits comfortably.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED

On a fresh/clean start the standard2 Log Scan test intermittently fails
"Should Not Find Forbidden" with:

  pods cert-manager-cainjector-... is forbidden: error looking up service
  account cert-manager/cert-manager-cainjector: serviceaccount ... not found

Investigation: the cainjector/controller/webhook Deployments and their
ServiceAccounts are created dynamically by the cert-manager operator, not
by MicroShift's static manifests. The kube-controller-manager ReplicaSet
controller can briefly attempt to create a pod before its ServiceAccount is
observed, logging this transient "forbidden" and retrying it away once the
SA lands. The workloads become ready (MicroShift healthcheck passes), so
this is a benign eventual-consistency startup race, not a MicroShift
manifest-ordering bug — the ordering is the operator's, not ours.

Add a scoped known-exceptions allowlist to the journalctl log-scan helper
and register this single pattern, so genuine "forbidden" regressions still
fail while this benign race is ignored.

While here, make the test itself easier to read: rename the per-boot
keyword to "Boot And Scan Journal", move the repeated journal-cursor
capture into it, and split the assertions into "Scan Boot Journal". The
two calls now read as "clean first boot" (forbidden check skipped, since a
clean boot logs the benign race above) and "restart" (must be forbidden-free).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED

Two release-scenario tests raced asynchronous resource creation:

- otp-workloads/statefulset-pvc: `oc wait pod/hello-statefulset-0
  --for=condition=Ready` ran immediately after creating the StatefulSet,
  before its controller created pod-0, so `oc wait` on the named pod
  failed with NotFound. Wait for the pod to exist first (reusing
  Wait Until Resource Exists) before checking readiness.

- ai-model-serving/ai-model-serving-online: the ServingRuntime was
  applied with a bare `oc apply` before the kserve validating webhook had
  endpoints, so the apply was rejected ("no endpoints available for
  service kserve-webhook-server-service"). Retry the apply until the
  webhook is serving, mirroring the retry already used for the
  InferenceService rollout.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

pre-commit.check-secrets: ENABLED